19 Covariance Matrices

Recall last note, Covariance matrix is defined as $$\mathrm{Cov}(\overrightarrow{X})=\mathbb{E}[(\overrightarrow{X}-\overrightarrow{\mu })(\overrightarrow{X}-\overrightarrow{\mu }{)}^{\mathrm{T}}],\overrightarrow{X},\overrightarrow{\mu }\in {\mathbb{R}}^n.$$
For all fixed ${\displaystyle \overrightarrow{v}\in {\mathbb{R}}^n}$, $$\begin{array}{rl}{\overrightarrow{v}}^{\mathrm{T}}\mathrm{Cov}(\overrightarrow{X})\overrightarrow{v}=& \mathbb{E}[{\overrightarrow{v}}^{\mathrm{T}}(\overrightarrow{X}-\overrightarrow{\mu })(\overrightarrow{X}-\overrightarrow{\mu })\overrightarrow{v}]=\mathbb{E}[Y^2]\ge 0,\end{array}$$
here ${\displaystyle Y={\overrightarrow{v}}^{\mathrm{T}}(\overrightarrow{X}-\overrightarrow{\mu })\in \mathbb{R}}$.

Thus covariance matrix must be positive semi-definite.

Let ${\displaystyle \mathrm{\Sigma }}$ be a positive semi-definite matrix, and let ${\displaystyle A}$ be its square-root matrix. Then ${\displaystyle \mathrm{\Sigma }=\mathrm{Cov}(\overrightarrow{X})}$, where ${\displaystyle \overrightarrow{X}=A\overrightarrow{Z}}$ and ${\displaystyle \overrightarrow{Z}\sim N_n(\overrightarrow{0},{\mathrm{\Sigma }}_n)}$.
More generally,

• ${\displaystyle \mathrm{\Sigma }}$ is a covariance matrix if and only if ${\displaystyle \mathrm{\Sigma }}$ is positive semi-definite.
• ${\displaystyle \mathrm{\Sigma }}$ is a non-singular covariance matrix if and only if ${\displaystyle \mathrm{\Sigma }}$ is positive definite.

When ${\displaystyle \mathrm{\Sigma }}$ is non-singular, ${\displaystyle \mathrm{\Sigma }}$ is invertible. By the proof we have ${\displaystyle \mathrm{\Sigma }=Q\mathrm{\Lambda }{Q}^{\mathrm{T}}}$, so$${\mathrm{\Sigma }}^{-1}=Q{\mathrm{\Lambda }}^{-1}{Q}^{\mathrm{T}}=Q{\mathrm{\Lambda }}^{-\frac{1}{2}}{\mathrm{\Lambda }}^{-\frac{1}{2}}{Q}^{\mathrm{T}}=\underset{{\mathrm{\Sigma }}^{-\frac{1}{2}}}{\underbrace{Q{\mathrm{\Lambda }}^{-\frac{1}{2}}{Q}^{\mathrm{T}}}}\underset{{\mathrm{\Sigma }}^{-\frac{1}{2}}}{\underbrace{Q{\mathrm{\Lambda }}^{-\frac{1}{2}}{Q}^{\mathrm{T}}}}.$$
Where ${\displaystyle {\mathrm{\Lambda }}^{-1}=\mathrm{diag}\{{\lambda }_1^{-1},\cdots ,{\lambda }_n^{-1}\},{\mathrm{\Lambda }}^{-\frac{1}{2}}=\mathrm{diag}\{{\lambda }_1^{-\frac{1}{2}},\cdots ,{\lambda }_n^{-\frac{1}{2}}\}}$.
So if ${\displaystyle \overrightarrow{x}\sim N_n(\overrightarrow{\mu },\mathrm{\Sigma })}$, where ${\displaystyle \mathrm{\Sigma }}$ is positive definite, then we can do standardization $$\overrightarrow{Z}={\mathrm{\Sigma }}^{-\frac{1}{2}}(\overrightarrow{X}-\overrightarrow{\mu })\sim N_n(\overrightarrow{0},I_n).$$

MGF

• 1-dim: ${\displaystyle X\sim N(\mu ,{\sigma }^2)}$. ${\displaystyle t\in \mathbb{R},M_X(t)=\mathbb{E}[{\mathrm{e}}^{tX}]={\mathrm{e}}^{\mu t+\frac{1}{2}{\sigma }^2{t}^2}}$.
• ${\displaystyle n}$-dim: ${\displaystyle \overrightarrow{X}\sim N_n(\overrightarrow{\mu },\mathrm{\Sigma }),\overrightarrow{t}\in \mathbb{R},M_{\overrightarrow{X}}(\overrightarrow{t})=\mathbb{E}[{\mathrm{e}}^{{\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X}}]={\mathrm{e}}^{{\overrightarrow{\mu }}^{\mathrm{T}}\overrightarrow{t}+\frac{1}{2}{\overrightarrow{t}}^{\mathrm{T}}\mathrm{\Sigma }\overrightarrow{t}}}$.
  ${\displaystyle {\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X}}$ is a univariate normal RV, with$$\begin{array}{rl}& \mathbb{E}[{\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X}]={\overrightarrow{t}}^{\mathrm{T}}\mathbb{E}[\overrightarrow{X}]={\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{\mu }.\\ & \mathrm{Var}({\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X})=\mathrm{Cov}({\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X},{\overrightarrow{t}}^{\mathrm{T}}\overrightarrow{X})={\overrightarrow{t}}^{\mathrm{T}}\mathrm{Cov}(\overrightarrow{X})\overrightarrow{t}\end{array}$$

Multivariate normal & ${\displaystyle {\chi }^2}$ distribution

The above result has many applications in Statistics.
E.g., ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }N(\mu ,{\sigma }^2)}$. ${\displaystyle S_n=X_1+\cdots +X_n}$. Then, the above result can be used to show$$\frac{1}{{\sigma }^2}\sum _{i=1}^n{(X_i-\frac{S_n}{n})}^2\sim {\chi }_{n-1}^2.$$

Multivariate CLT

(Recall 1-dim CLT.)

• 1-dim: ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }\mathrm{Bernoulli}(p)}$, then$$\sqrt{n}\left(\frac{\frac{S_n}{n}-p}{\sqrt{p(1-p)}}\right)\stackrel{\mathrm{d}}{\to }N(0,1),n\to \mathrm{\infty }.$$
• ${\displaystyle k}$-dim: ${\displaystyle \mathbb{P}(X=j)=p_j}$, ${\displaystyle \sum _{i=1}^n\mathbf{1}\{X_i=j\}=C_{n,j}}$, ${\displaystyle {\overrightarrow{C}}_n=(C_{n,1},\cdots ,C_{n,k}{)}^{\mathrm{T}}}$ is a random vector with ${\displaystyle \mathbb{E}[C_{n,j}]=n{p}_j}$.

Fact: ${\displaystyle M}$ is idempotent and symmetric with ${\displaystyle \mathrm{rank}(M)=k-1}$. (Apply the theorem. )